DOCUMENT RESUME 



ED 392 828 



TM 024 506 



AUTHOR 

TITLE 

PUB DATE 
NOTE 

PUB TYPE 



Marshak, David 

District*"Based Program Assessment: One Way To Create 
’’Schools That Learn. ” 

95 

lOp. 

Reports - Evaluative/Feasibility (142) 



EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



MFOl/PCOl Plus Postage. 

Data Collection; Educational Assessment ; ’'Educational 
Improvement; Elementary Secondary Education; 
’"'Evaluation Methods; Intellectual Disciplines; 
Principals; Program Evaluation; ’"^Rural Schools; 
’"'School Districts; ’"^Sciences; State Programs; 
Teachers; Testing Programs 
’"^Vermont 



ABSTRACT 

Twelve teachers and two principals from the Addison 
Northeast Supervisory Union School District (Vermont) met to consider 
what a small rural school district can do to assess its own programs 
and to discuss how to go about program assessment with the goal of 
helping schools become effective learning organizations. The district 
serves about 1,800 students in 5 elementary schools, a middle school, 
and a high school. Participants began by defining a common vocabulary 
of program assessment and preparing a mission statement. In defining 
their goals, the educators decided on assessment of the major 
disciplines, beginning with science. Evaluation questions were 
formulated in the assessment categories of inputs, implementation, 
and outcomes; and participants began to look for data about each 
question. Test results from district and state assessments, including 
the Vermont writing and mathematics portfolios, were acknowledged as 
an important source of data. The participants developed a Science 
Assessment Plan with data collection for a 4-year cycle, followed by 
an assessment committee's evaluation of the data. The experiences of 
this district demonstrated that a small district can establish its 
own evaluation plan, but that it probably cannot develop all the 
tools needed for evaluation on its own. An appendix presents the 
program assessment mission statement. (SLD) 
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What can a smaii, rurai pubiic school district do to assess its own programs? 
How does a district begin and undertake such an effort? And can district-based 
program assessment help schools to become effective learning organizations? 
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In late June of 1989 after the end of the school year, I met for three days with 
twelve teachers and two principals from Addison Northeast Supervisory Union schools 
to begin to answer these questions. Addison NE, in Bristol, Vermont and four 
surrounding farm and mountain towns, includes five elementary schools and a middle 
school program and grades 9-12 high school housed in Mt. Abraham Union High. The 
district serves about 1800 students. Each elementary school was represented by at 
least one teacher, and the secondary programs were represented by two science 
teachers, a math teacher, and an English teacher. Two elementary principals were 
also active participants in our group. I was the district’s four-fifths time Curriculum and 
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Assessment Coordinator. 

I began the process by introducing a number of concepts to the team, so that 
we could employ a common vocabulary. These included the terms program (the 
district’s K-12 curriculum and the actuality of the instructional implementation of that 
curriculum), program assessment (identifying questions about student learning, 
gathering data from which to generate answers to those questions; making sense of 
the data and articulating conclusions about the data), inputs (what goes into the 
system of the school and classroom to create and support the program), 
implementation (teaching and learning activities), and outcomes (what students 
know and can do as a result of the program). We explored each term in detail and 
distinguished particularly between program assessment and other kinds of evaluation, 
for example, evaluation of individual students or teachers. 

Once we had developed a common understanding of program assessment, 
our next step involved the articulation of a program assessment mission statement. 
Through a series of small group brainstorming exercises, whole group discussion, and 
small group writing, we generated a draft that evolved over our three days of work 
sessions (see the final page of this article). 

We devoted an hour to discussing how to organize our first steps toward 
program assessment. Although there was some sentiment expressed in favor of a 
multidisciplinary or integrative assessment, we moved fairly quickly to agreement on a 
disciplinary focus. Addison NE had just completed the third year of what was to be a 
complete curriculum renewal. We had written new science and math curricula for 
grades K-12 first, which we had begun to implement; science first in 1987-88, followed 
by mathematics in 1988-89. Given that commitment, we agreed that it made sense to 
assess each major discipline independently, starting with science. 

With these parameters set, we focused the remainder of our time on the 
following question; “What is it that we want to know about our district science program 



in terms of inputs, implementation, and outcomes?" We divided into three working 
groups, with each group focusing on one key category of assessment. Over the last 
day and a half, we generated dozens of questions, considered most of these, and 
collaboratively honed our list of key questions to nine for each category. Examples of 
these questions include the following: 



INPUTS 

Are adequate funds available for the science program? 

Do the district curriculum guidelines reflect current recommendations of the 
valid professional research related to science curriculum and instruction? 

Are new teachers hired knowledgeable about child development and in tune 
with the goals of the science program? 



IMPLEMENTATION 

Do the teaching and learning activities support the goals of the science 
curriculum? 

Are the teaching and learning activities reflective of: (a) the balance between 
physical and natural science: (b) the balance between required topics and 
those based on teacher and student selection; (c) thorough instruction of 
process skills and concepts/content? 



OUTCOMES 

Can students use science process skills to solve problems? 

Does the science program achieve the following motivational and attitudinal 
goals: (a) students’ pursuit of individual interests in science; and (b) students’ 
expression of positive attitudes toward science? 

Does the science program promote students’ fulfillment of their own post- 
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graduate expectations and those of the community for them? 

How do various sub-groups of students (i.e., gender, SES) perform in relation 
to the outcome goals of the science program? 

In the fall of 1989, 1 enlisted almost all of the participants from the June work 
session as members of the district Program Assessment Committee. With a 
combination of after-school meetings and a few released half days, we moved ahead 
in developing the Science Program Assessment. For each question that we had 
agreed to incorporate in the assessment process, we needed to find or create a means 
for generating data that responded to that question. A few questions could be 
answered by the standardized CTBS tests that we already used, particularly once we 
learned to conduct item analyses of the various tests, so we could discern which ones 
matched our curricular goals, and to disaggregate the scores for various groups of 
students. For some questions we drafted questionnaires, which required field testing 
and revision. Later in the year we administered these questionnaires to teachers and 
students at several grade levels. 

Other questions required us to search for data that was already gathered 
elsewhere in the district. Members of the Committee learned how to analyze 
documents generated by professional organizations, the state, and the district. We 
also invented an assessment method called the “science folder," in which teachers 
collected specified numbers of samples of their own instructional planning and of 
student work. At the end of the school year we developed a rubric for scoring these 
folders in relation to most of our implementation questions. 

Our committee work on program assessment in 1989-90 had two themes. 
Although we had cut our lists of assessment questions relentlessly at our June 
session, we discovered throughout the next school year that we still had too many 
questions to follow, given our limited work time and resources. So we continued to 



pare down the list in each category, finally arriving by the end of the year at seven 
input questions, three implementation questions, and eight outcome questions. By 
May 1990 we were convinced that these were a realistic number. 

The second them£> was double-edged. We had accomplished a great deal in 
developing assessment methods based on questionnaires, artifact analysis, and the 
use of already gathered data. But our efforts to develop performance-based 
assessments were stymied by our lack of time and resources. 

The Assessment Committee met again for three days at the end of June 1990. 
We scored science folders, analyzed data, and conceptually organized our first District 
Assessment Report, which I would write in September. We also drafted a Mathematics 
Program Assessment plan, based directly on the Science Plan. 

In the fall of 1990 we were fortunate to gain access to the Manipulative Skills 
Test of the New York State Elementary Science Program Evaluation Test (ESPET), a 
performance-based assessment for 4th graders in science process skills. The skills 
tested by the ESPET matched our science curriculum at 100%. In this school year, we 
included the ESPET in our assessment efforts and administered the 45 minute hands- 
on manipulative test to all 4th graders and a sample of 6th graders in our schools. 

In the 1991-92 school year Vermont initiated a state-wide Writing Portfolio and 
Math Portfolio system. We integrated the Math Portfolio into our Math Program 
Assessment Plan. We also finally had the time and resources to engage in a 
comprehensive item analysis of the CTBS tests in all subject areas, with a focus on the 
degree of match between our district curricula and the CTBS test items. We examined 
the tests at grades 4, 6, and 9 and discovered high match levels for all science tests, a 
low match for the math test at grade 4 but higher levels of match at grades 6 and 9. 
These item analysis results will help district administrators and teachers make 
decisions in the future about how and when to administer the CTBS tests: no longer as 
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an isolated, single, and partial indicator of outcomes but as an integral element in a 
coherent program assessment plan. 

In 1992-93 Addison NE conducted the fourth year of the Science Assessment 
Plan. What the plan calls for is data collection for a four year cycle, to be followed by 
an Assessment Committee evaluation of the four years of data. The Assessment 
Committee members met in the fall of 1993 to consider the four years of data and 
articulate their conclusions about the science program that flow from the data. Partly as 
a result of this analysis, the Committee has recommended the articulation of an 
integrated science, math, and technology curriculum for the district. 

In 1993-94 the district began work on a Language Arts Program Assessment 
effort, including the Vermont Writing Portfolio. The district also followed through on the 
fourth year of the Math Program Assessment effort and began the first year of a new 
cycle of its Science Program Assessment. These various program assessment 
activities have continued through this current school year, the sixth year of their 
enactment, and their value to the district is sufficiently evident that they have survived a 
turnover in the offices of Superintendent and Curriculum and Assessment Coordinator. 

In June 1993 the Regional Laboratory for Educational Improvement in the 
Northeast & Islands recognized the Addison Northeast District Math and Science 
Program Assessment as a “Program of Excellence." A description of the Assessment 
Program is included in the Lab’s publication Best Practices in Mathematics and 
Science Education. 




Here are the conclusions that we can share from five years of experience with 
this process. Can a small, rural public school district enact its own program 
assessment successfully? Yes! Can such a district then base its future curriculum 
development and instructional innovation on what has been learned from this 
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assessment? We believe so. Addison NE reached this stage in that process during 
the 1993-94 schooi year and began to modify its science and math curricuia and 
instruction partiy as a resuit of program assessment data and conciusions. Can such a 
district deveiop aii the toois it needs to accompiish its program assessment? No. The 
district needs to draw on and incorporate all the appropriate assessment tools that it 
can gather: from its own state department of education, from other schools and states, 
from publishers, and from university resources. 

How has the program assessment effort affected teachers, administrators, 
school board members and the public, and students? With the leadership of the 
Curriculum and Assessment Coordinator, the teachers on the Assessment Committee 
have developed and conducted all of the assessment activities. These teachers have 
played a central role in explaining program assessment to their colleagues and in 
engaging collegial support and involvement. Over the years teachers in the district 
have become increasingly interested in receiving program assessment data, in 
considering its meanings in relation their own teaching, and in modifying their 
instruction as a result of this consideration. The program assessment effort has also 
begun to build a system of accountability that most teachers own and value, because 
teachers were its primary architects. 

Administrators and school board members have grown increasingly interested 
in learning about and weighing the assessment outcomes. We have also published a 
District Assessment Report each year, included assessment outcomes in newsletters 
to the public, and involved citizens in discussions of assessment data at our annual 
School Report Night and in many other community forums. 

What is the value for students? That value is unfolding. The more we know 
precisely how well we are achieving our goals for student learning, the more we will 
be able both to celebrate our accomplishments and to address our failures. We can 
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only mitigate inadequacies in our classrooms if we know what they are. Focused, 
rigorous program assessment can help to provide us with the information we need to 
create schools where all children can succeed. 

Our experience in Addison NE suggests that focused, rigorous district-based 
program assessment, generated and conducted by teachers as well as administrators, 
can help to develop school cultures where continuous improvement is the watchword 
not only for students but also for educators. Our experience has also shown us that the 
values ” commitments to inquiry, data, and reflection, and to school improvement -- 
and competencies -- asking productive and challenging questions, collecting rich data 
in response to those questions, analyzing and judging the data, and innovating based 
on those judgments -- that drive successful program assessment are some of the 
same values and competencies that create an effective learning organization. 
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ADDISON NORTHEAST 

PROGRAM ASSESSMENT MISSION STATEMENT 
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The mission of program assessment in Addison NE is to increase the quantity and improve 
the quality of student learning and growth in our schools. Assessing our subject-area programs will 
accomplish this mission by providing the following information relating to each subject-area 
program; 

- the extent to which inputs are of high quality as well as adequate and appropriate; 

" the extent to which the curriculum is being implemented effectively (implementation); and 
" the extent to which the stated goals for student learning are being achieved (outcomes). 

This information will facilitate the accomplishment of the following; 

- the evaluation of each subject-area curriculum and program in terms of inputs, teaching and 
learning activities (implementation), and outcomes; 

- the establishment of accountability to students, parents, other community members, and district 
educators in relation to Addison NE programs; 

" the informing of school boards, parents, and community members of program successes and 
limitations; 

- the accomplishment of needed changes in curriculum and/or program as indicated by 
assessment data; and 

” the generation of data to support budgetary and program requests for improving programs. 

Program assessment activities should; 

- be positive and engaging to those involved, and realistic in scope in terms of time and energy; 

” be as compatible as possible with teaching and learning activities; 

- be as inherently educative and interesting for students as possible; and 

- provide useful information to teachers in an ongoing and formative manner as well as a 
summative manner. 
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